Over the past 30 years, the problems for effective software development have been well studied. Some have approached the problem from the language viewpoint by developing rich semantics which encapsulate a high degree of well-tested functionality, thereby accentuating reuse. Some have approached the problem from an organizational viewpoint, and have focused on the phases of software development within an organization and how to manage software development effectively.
While all of these approaches resulted in contributions to enhanced software development productivity and quality, as yet there are no fail-safe techniques which render the phases of testing and debugging obsolete. If anything, these improvements have made software development more difficult to conduct and raised the importance of quality assurance. It has become more difficult due to the emergence of more complicated and richly semantic development languages such as C++, and by the increased complexity in applications with the introduction of parallel and distributed programming techniques. These enhancements have increased the importance of quality due to the increased reliability demands placed on vended component software in conjunction with the increasing competitiveness in the market for quality software. Many of the problems behind production software are inherent in the testing and debugging phases of a project. Usually software development is depicted as proceeding from the program routine (unit testing) level to a coherent sub-program (component testing) level to the full program (integration testing) level. In fact, this testing fluidity is the exception in practice. In many cases, logic testing at the unit level cannot be satisfactorily achieved until final integration. This is due to the fact that the complexity or quantity of information required to adequately drive unit testing cannot be exercised without invoking a suite of software larger than the unit but smaller than the integration.
A common approach to dealing with this problem is to build additional software that can in some manner fabricate input data for a given program unit. This technique, called "scaffolding", does aid in the testing and development cycle in exercising unit programs. It has several drawbacks however. One is that variation in the complexity of test data is usually limited. This is mainly a matter of practicality. Due to the fact that building scaffolding is a "detour" from the project's development and typically does not warrant priority for that flexibility, resources are less likely to be spent on scaffolding than on the project's program. The bigger problem is that building scaffolding augments development costs. As such, it is important to cost accounting and schedules to reduce this expense as much as possible--especially since the scaffolding program is not properly part of the product and essentially is "throw-away".
In large part, the problems described above are exacerbated by the fact that programming tools are not flexible enough to facilitate testing through the various stages of the programming cycle. Tools such as debuggers generally require that the application be built fully, if not partially with scaffolding, in order to be applied to problem resolution. In many ways this is due to a debugging tool designed to runtime requirements. That is, programming tools are designed for execution environments, which to operate reasonably need to be complete and consistent.
The upshot of all these scenarios is that project uncertainties and anxieties are pushed to the later development phases. It is only by putting all the pieces of an application together that a realization and understanding of all its problems emerges.